Using storage access statistics to determine mirrored extents to migrate from a primary storage system and a secondary storage system to a third storage system

ABSTRACT

Provided are a computer program product, system, and method for using storage access statistics to determine mirrored extents to migrate from a primary storage system and a secondary storage system to a third storage system. A determination is made of access statistics with respect to mirrored extents of data at the primary storage mirrored to the secondary storage to migrate to the third storage. A first set of the mirrored extents associated with access statistics indicating a highest level of access of the mirrored extents are migrated from the secondary storage to the third storage. A second set of the mirrored extents associated with access statistics indicating a lower level of access than the mirrored extents in the first set are migrated from the primary storage to the third storage.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a computer program product, system, andmethod for using storage access statistics to determine mirrored extentsto migrate from a primary storage system and a secondary storage systemto a third storage system.

2. Description of the Related Art

Data backup systems can provide continuous availability of productiondata in the event of a sudden catastrophic failure at a single point intime or data loss over a period of time. In one such disaster recoverysystem, production data is replicated from a primary storage system to asecondary storage system. Different data replication technologies may beused for maintaining remote copies of data at a secondary site, such asInternational Business Machine Corporation's (“IBM”) Metro Mirror Peerto Peer Remote Copy (PPRC), Extended Remote Copy (XRC), Coupled XRC(CXRC), Global Copy, and Global Mirror.

Data mirrored between and stored in a primary storage system andsecondary storage system may be migrated to remote cloud storage andthen recalled when later needed. Data mirrored in a synchronousreplicated environment may be migrated from and recalled to the primaryserver in the mirror relationship. When recalling data in cloud storageto the primary storage server, the recall is not complete until therecalled data is migrated back to the secondary storage server. One suchprogram for migrating and recalling mirrored data is InternationalBusiness Machine Corporation's Transparent Cloud Tiering product.

There is a need in the art for improved techniques for migrating andrecalling mirrored data stored in primary and second storage systemswith respect to a remote storage, such as remote cloud storage.

SUMMARY

Provided are a computer program product, system, and method for usingstorage access statistics to determine mirrored extents to migrate froma primary storage system and a secondary storage system to a thirdstorage system. A determination is made of access statistics withrespect to mirrored extents of data at the primary storage mirrored tothe third storage to migrate to the third storage. A first set of themirrored extents associated with access statistics indicating a highestlevel of access of the mirrored extents are migrated from the secondarystorage to the third storage. A second set of the mirrored extentsassociated with access statistics indicating a lower level of accessthan the mirrored extents in the first set are migrated from the primarystorage to the secondary storage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a data replication environment.

FIG. 2 illustrates an embodiment of components in a server in thereplication environment of FIG. 1.

FIG. 3 illustrates an embodiment of access statistics used to determinewhether to select the primary storage sever or secondary storage serverto migrate and recall mirrored extents.

FIG. 4 illustrates an embodiment of operations to select the primarystorage server or secondary storage server to migrate mirrored extentsto the remote storage.

FIG. 5 illustrates an embodiment of operations to determine an aggregateactivity score for mirrored extents based on multiple access statistics.

FIG. 6 illustrates an embodiment of operations to select the primarystorage server or secondary storage server to recall mirrored extentsfrom the remote storage.

FIG. 7 illustrates an embodiment of path statistics used to determineservers to use for recall operations.

FIG. 8 illustrates an embodiment of operations to determine whethermirror paths are degraded.

FIG. 9 illustrates a computing environment in which the components ofFIG. 1 may be implemented.

DETAILED DESCRIPTION

To migrate mirrored data to remote cloud storage and recall the mirroreddata from the remote call storage, the migration operations may be splitbetween the primary storage server and secondary storage server. If boththe primary and secondary storage servers are involved in recallingmigrated data, then traffic on the mirror paths is reduced becauserecalled data does not have to be mirrored to the secondary storageserver.

Described embodiments provide improvements to computer technology formigrating data using both the primary and secondary storage server byconsidering access statistics for sections of the primary and secondarystorages including the data extents to migrate. The access statisticsare used to have the primary storage server, which is usually theproduction server receiving host Input/Output (“I/O”) requests, handlemigration for mirrored extents that are on sections of the primarystorage system experiencing a relatively lower level of access on theprimary storage system and have the secondary storage system handle themigration for mirrored extents that are on sections of the primarystorage experiencing a relatively higher level of access. This techniquereduces the processing burdens on the primary storage server because theprimary storage server avoids having to migrate mirrored data withrespect to ranks or sections of the primary storage that are alreadyexperiencing a high level of host access. The secondary storage server,which is typically not handling host I/O requests, may handle themigration of mirrored data on storage sections of the primary storageserver experiencing a high level of access to reduce processing burdensfor high access regions of the primary storage.

Described embodiments further provide improvements to computertechnology for recalling data using the primary and secondary storageservers by considering path statistics for mirror paths used to mirrordata between the primary storage server and the secondary storageserver. The described embodiments determine whether the mirror paths aredegraded and, if so, select to use both the primary and secondarystorage servers to separately call data from the third storage to avoidhaving to mirror recalled data on the degraded mirror paths if only theprimary storage server is used for migration In making the determinationof sufficiently degraded performance, the migration/recall manager mayconsider degradation criteria indicating degradation, such as bandwidthutilization, high response time, unacceptable ratio of successfultransfers, and high failure rate mode, in the mirror paths. If themirror paths are not degraded, then the data may be recalled by only theprimary storage server and then mirrored to the secondary storage serverover the mirror paths.

FIG. 1 illustrates an embodiment of a data replication environmenthaving a primary storage system 100 _(P) including a primary storageserver 200 _(P) managing access to a primary storage 102 _(P) andsecondary storage system 100 _(S), each including storage servers 200_(P) and 200 _(S), respectively, managing access to volumes 104 _(P) and104 _(S) configured in storages 102 _(P) and 102 _(S). One or more hosts200 _(H) may direct I/O requests to the primary storage system 100 _(P)or secondary storage system 100 _(S), where the primary storage system100 _(P) may comprise a production storage to which most I/O requestsare directed, and the secondary storage system 100 _(S) is used mostlyfor mirroring data in case of a failover. The primary storage server 200_(P) may mirror data in the primary volumes 104 _(P) to the secondarystorage system 100 _(S), also referred to a secondary storage or targetstorage, to maintain data in consistency groups at the second storageserver 200 ₂. Data may be mirrored synchronously, such that a write tothe primary storage server 200 _(P) is not considered complete until thedata is transferred to the secondary storage server 200 _(S). The datamay be mirrored over a first network 110 having paths mirror formigrating data between the primary storage system 100 _(P) and thesecondary storage system 100 _(S).

Data in the volumes 104 _(P), 104 _(S) may be configured in ranks, wherea rank is comprised of multiple extents, and wherein each extent iscomprised of numerous tracks. Other storage units may be used thanranks, extents, and tracks. For instance, the volumes may compriselogical devices or drives configured in sections or partitions ofstorage other than ranks, that are comprised of groups of blocks ortracks. The term “rank” as used herein may refer to any section or areaof storage having groups of tracks or blocks, referred to as extents.

The primary storage server 200 _(P) and the secondary storage server 200_(S) may each migrate mirrored extents of data in the volumes 104 _(P)and 104 _(S) to a remote cloud storage 112 over a second network 114.The remote cloud storage 112 may comprise a cloud storage systemprovided by a cloud storage service provider. Examples of cloud storage112 service providers include DropBox®, Google® Drive, Amazon CloudDrive®, Amazon® S3, IBM® Cloud Object Storage System™, etc. (Dropbox isa registered trademark of Dropbox, Inc., Google is a registeredtrademark of Google, Inc., Amazon and Amazon Cloud Drive are trademarksof Amazon Technologies, Inc.; and IBM and Cloud Object Storage Systemare trademarks of IBM throughout the world).

The term “storage system” as used herein may refer to a storage server200 _(P), 200 _(S) and/or the storage 102 _(P), 102 _(S) managed by theserver. The term “server” or “storage server” may be used to refer tothe servers 200 _(P), 200 _(S)

The storages 102 _(P), 102 _(S) may comprise different types or classesof storage devices, such as magnetic hard disk drives, solid statestorage device (SSD) comprised of solid state electronics, EEPROM(Electrically Erasable Programmable Read-Only Memory), flash memory,flash disk, Random Access Memory (RAM) drive, storage-class memory(SCM), etc., Phase Change Memory (PCM), resistive random access memory(RRAM), spin transfer torque memory (STM-RAM), conductive bridging RAM(CBRAM), magnetic hard disk drive, optical disk, tape, etc. The volumes104 _(P), 104 _(S) may further be configured from an array of devices,such as Just a Bunch of Disks (JBOD), Direct Access Storage Device(DASD), Redundant Array of Independent Disks (RAID) array,virtualization device, etc. Further, the storages 102 _(P), 102 _(S) maycomprise heterogeneous storage devices from different vendors anddifferent types of storage devices, such as a first type of storagedevices, e.g., hard disk drives, that have a slower data transfer ratethan a second type of storage devices, e.g., SSDs.

The first network 110 used by the storage systems 100 _(P) and 100 _(S)to mirror data may comprise mirror paths configured in a storage networksuch as one or more interconnected Local Area Networks (LAN), StorageArea Networks (SAN), Wide Area Network (WAN), peer-to-peer network,wireless network, etc. The second network 114 may comprise a networkaccessible to a remote cloud storage 112, such as the Internet, a WideArea Network (WAN). In alternative embodiments, the first 110 and second114 networks may be the same network. For instance, the remote cloudstorage 112 may comprise a third storage coupled to the first network110.

FIG. 2 provides an embodiment of components of a server 200 _(i)involved in data mirroring, such as the primary storage server 200 _(P),secondary storage server 200 _(S), and host 200 _(H). The server 200_(i) includes a processor 202 and a memory 204 including programsexecuted by the processor 202 as well as a cache 206 to cache read andwrite data for the first storage 102 _(i). A portion of the cache 206may also be used to mirror data in a consistency group.

The memory 204 includes an operating system 208, which configures andmanages volumes in attached storage and maintains volume tables 210,such as a volume table of contents (VTOC), file allocation table, etc.,providing information on the configured volumes 104. The operatingsystem 208 further manages I/O requests with respect to the volumes 104_(i).

The memory 204 includes a copy manager 212 to create and manage mirrorcopy relationships 214 of source data extents in primary volumes 104_(P) in the primary storage system 100 _(P), also referred to as sourcestorage, to target data extents in the secondary storage system 100_(S), also referred to as the target storage, as part of consistencygroups. In one embodiment, the primary storage system 100 _(P) may havethe source storage and the secondary storage system 100 _(S) may havethe target storage of mirror copy relationships to mirror source volumesor other data units to corresponding target volumes or data units. Thecopy manager 212 may mirror extents of tracks in the primary volume 104_(P) synchronously to a secondary volume 104 _(S) in the secondarystorage 102 _(S) over mirror paths in the first network 100. Differentdata replication technologies may be used for the copy manager 212 tomaintain remote copies of data at the secondary storage system 100 _(S),such as International Business Machine Corporation's (“IBM”) MetroMirror Peer to Peer Remote Copy (PPRC), Extended Remote Copy (XRC),Coupled XRC (CXRC), Global Copy, and Global Mirror Copy, includingprograms provided by other vendors.

The server 200 _(i) includes one or more storage adaptors 216 tocommunicate with devices in the storage 102 and one or more networkadaptors 218 to communicate with the networks 110 and 114.

The server 200 _(i) further includes a migration/recall manager 220 tomanage the migration and recall of extents from the primary 102 _(P) andsecondary 102 _(S) storages to the remote cloud storage 112 or otherremote storage. For instance, the migration/recall manager 220 maycreate backup objects including mirrored extents mirrored between theprimary storage 102 _(P) and secondary storage 102 _(S) to store orarchive in the remote cloud storage 112. The migrated backup objects maybe recalled from the remote cloud storage 112 to unpack and store theextents therein in the primary 102 _(P) and secondary 102 _(S) storages.

The server 200 _(i) may further maintain primary access statistics 300_(P) having information on I/O accesses to ranks configured in theprimary storage 102 _(P) and secondary access statistics 300 _(S) havinginformation on I/O accesses to ranks configured in the secondary storage102 _(S). The primary storage server 200 _(P) and secondary storageserver 200 _(S) may each gather access statistics 300 _(P) and 300 _(S),respectively, with respect to the primary storage 102 _(P) and secondarystorage 102 _(S), respectively. The primary storage server 200 _(P) andsecondary storage server 200 _(S) may transfer the access statistics 300_(P) and 300 _(S) they each gather to each other to maintain and useduring migration and recall operations. The server 200 _(i) furtherincludes path statistics 700 having information on bandwidth andperformance metrics for mirror paths formed in the first network 110.

The program components in the memory 204, including 208, 212, 220, areshown in FIG. 2 as program code loaded into the memory 204 and executedby the processor 202. Alternatively, some or all of the componentsfunctions may be implemented in hardware devices, such as in ApplicationSpecific Integrated Circuits (ASICs), Field Programmable Gate Array(FPGA) or executed by separate dedicated processors.

The memory 204 may comprise one or more memory devices volatile ornon-volatile, such as a Dynamic Random Access Memory (DRAM), a phasechange memory (PCM), Magnetoresistive random-access memory (MRAM), SpinTransfer Torque (STT)-MRAM, SRAM storage devices, DRAM, a ferroelectricrandom-access memory (FeTRAM), nanowire-based non-volatile memory, andNon-Volatile Direct In-Line Memory Modules (DIMMs), NAND storage, e.g.,flash memory, Solid State Drive (SSD) storage, non-volatile RAM, etc.

FIG. 3 illustrates an embodiment of access statistics 300 gathered for avolume, such as the primary 104 _(P) and secondary 104 _(S) volumes, andincludes a storage system/volume 302 for which the statistics aregathered; the rank 304 or other section or grouping of data units, e.g.,tracks, blocks, in the storage to which the statistics apply; a readlatency 306 indicating an average time to process read requests to therank 304; a read activity 308 indicating an amount of read activitytoward the rank 304, such as throughput of reads, e.g., in megabytes persecond, I/O Operations (IOPSs) per second, etc.; and an I/O activity 310toward the rank 304, such as the combination of destage and stageoperations toward tracks in the rank 304.

FIG. 4 illustrates an embodiment of operations performed by themigration/recall manager 220 in the primary server 200 _(P) and/orsecondary server 200 _(S) to perform migration of mirrored extentsspecified in a migration command or part of a scheduled migrationoperation. Upon processing (at block 400) a migration request to migratemirrored extents at the primary storage system 100 _(P) and thesecondary storage system 100 _(S), the migration/recall manager 220determines (at block 402) whether the amount of data in the mirroredextents to migrate exceeds a threshold amount of data. If (at block 402)the threshold amount of data is not exceeded, then the migrated extentsare migrated (at block 404) from the primary storage sever 200 _(P) tothe remote cloud storage 112 via the second network 114. If (at block402) the mirrored extents exceed a size threshold but are not insynchronous replicated volumes, then control proceeds to block 404 tomigrate just from the primary storage 102 _(P). If (at block 406) theextents to migrate are in a synchronous replicated volume and exceed thethreshold size, then a determination is made (at block 408) of theprimary 300 _(P) and secondary 300 _(S) access statistics for themirrored extents at the primary 102 _(P) and secondary storages 102_(S), respectively, where a higher value indicates a higher level ofaccess.

In one embodiment, the access statistics 300 may be for a rank or othersection of the primary and secondary storages including the mirroredextents. The determined access statistics may comprise a read latency306, a read activity, or an I/O activity 310 with respect to differentranks to use to determine which mirrored extents to migrate from theprimary 100 _(P) or secondary 100 _(S) storage systems. In oneembodiment, the migration/recall manager 220 may consider only one ofthe types of access statistics 306, 308 or 310 indicating a level ofaccess with respect to the rank or section of storage including themirrored extent. In a further embodiment, the migration/recall manager220 may calculate an aggregate activity score based on two or more ofthe access statistics 306, 308, 310, such as an aggregate calculatedaccording to the operations of FIG. 5. In the described embodiments,access statistics are maintained for a rank or section of the primary orsecondary storage including the mirrored extent. In an alternativeembodiments, access extents may be maintained for smaller groups ofmirrored extents or for single extents or other groupings of data units,such as tracks, blocks, etc.

For each mirrored extent to migrate, the migration/recall manager 220may determine (at block 410) a delta of the primary access statistic andthe secondary access statistic that is being used, e.g., 306, 308 and/or310, for the rank 304 in the primary 102 _(P) and secondary 102 _(S)storages including the mirrored extent. The mirrored extents to migratemay then be sorted (at block 412) in a sorted list. The recall/migrationmanager 220 determines (at block 414) a first set of mirrored extents tomigrate having highest deltas (e.g., relatively greater level of accessat the primary storage than the secondary storage), such as a half ofthe sorted list with the highest deltas. A determination is made (atblock 416) of a second set of mirrored extents to migrate having lowestdeltas (e.g., relatively lower level of access at the primary storagethan the secondary storage), such as a half of the sorted list with thelowest deltas.

The recall/migration manager 220 may then send (at block 418) a commandto the secondary storage server 200 _(S) to migrate the first set of themirrored extents having highest level of access from the secondarystorage 102 _(S) to the remote cloud storage 112. The recall/migrationmanager 220 may further send (at block 420) a command to the primarystorage server 200 _(P) to migrate the second set of the mirroredextents from the primary storage 102 _(P) to the remote cloud storage112.

With the embodiment of FIG. 4, access 300 statistics are used to dividethe mirrored extents into a first group of mirrored extents stored insections or ranks having a relatively higher level of access at theprimary storage 102 _(P) than the secondary storage 102 _(S) than asecond group of mirrored extents. The described embodiments operate tohave the primary storage server 200 _(P), which has greater processingburdens than the secondary storage server 200 _(S), manage the migrationof mirrored extents on ranks of storage that are experiencing a lowerlevel of access at the primary storage 102 _(P). This reduces theprocessing burdens on the primary storage server of having to handlemigration of mirrored extents from ranks or sections of the primarystorage 102 _(P) already experiencing a high level of activity from hostI/O access. Having to handle host I/O access to extents on a rank whoseextents are being migrated adds an extra level of processing burden tothe primary storage server handling those operations.

For instance, the primary storage server 200 _(P) handles the migrationof mirrored extents on ranks having a lower delta, indicating arelatively low level of activity at the primary storage server, becausethe primary storage server 200 _(P) will experience a lower level ofhost access to those low delta ranks or sections of the primary storage102 _(P) having the mirrored extents during migration, which reduces theprocessing burdens of having to handle host I/O to mirrored extents thatare being migrated. Further, the secondary storage server 200 _(P)handles the migration of mirrored extents on ranks having a higherdelta, indicating a relatively higher level of activity at the primarystorage server 102 _(P), to save the primary storage server 200 _(P)from having to further burden an overloaded rank with the migration ofmirrored extents on a rank or section experiencing a high level of hostaccess. In this way, the primary storage server 200 _(P) is relieved ofthe burden of having to migrate those mirrored extents on ranks that areexperiencing the highest level of access at the primary storage server200 _(P). The primary storage server 200 _(P) will however handle themigration of the mirrored extents experiencing the lowest level ofaccess at the primary storage server 200 _(P), which can be migratedwith a lower likelihood of being interrupted by host I/O access.

FIG. 5 illustrates an embodiment of operations performed by themigration/recall manager 220 to calculate an aggregate activity scorebased on the read latency 306, read activity 308, and I/O activity 310to use to sort the mirrored extents for migration. Upon initiating (atblock 500) the operation to determine the aggregate activity score formirrored extents to produce the sorted list, the migration/recallmanager 220 performs the operations at blocks 502 through 512 for eachmirrored extent i to migrate. Determinations are made of: of a readlatency difference (at block 504) of read latency 306 of a rank 304 inprimary storage 302 including mirrored extent i in the primary accessstatistics 300 _(P) and read latency 306 of a rank 304 in a secondarystorage 302 including mirrored extent i in secondary access statistics300 _(S); a read activity difference (at block 506) of read activity 308of rank 304 in primary storage 102 _(P) including mirrored extent i inprimary access statistics 300 _(P) and read activity 308 of rank 304 insecondary storage 102 _(S) including mirrored extent i in secondaryaccess statistics 300 _(S); and a rank I/O activity difference of I/Oactivity 310 (e.g., stages and destages) at a rank 304 in primarystorage 102 _(P) including mirrored extent i in primary accessstatistics 300 _(P) and I/O activity 310 of rank 304 in the secondarystorage 102 _(S) including mirrored extent i in secondary accessstatistics 300 _(S). The migration/recall manager 220 determines (atblock 510) an aggregate activity score for mirrored extent i comprisinga weighted aggregate of read latency difference, read activitydifference, and rank I/O activity difference for mirrored extent i.Different ratios may be used to weight the read latency, read activity,and rank I/O differences.

With the embodiment of FIG. 5, multiple different access statistics maybe used to determine an aggregate activity score that reflects differenttypes of access statistics to provide a more overall measure of accessof ranks including mirrored extents to use to sort the mirrored extentsto determine how to divide mirrored extents to migrate between theprimary 200 _(P) and secondary 200 _(S) storage servers.

FIG. 6 illustrates an embodiment of operations performed by themigration/recall manager 220 in the primary server 200 _(P) and/orsecondary server 200 _(S) to recall mirrored extents from the remotecloud storage 112 in a recall command or part of a scheduled recalloperation. Upon processing (at block 600) a migration request to recallmirrored extents from the remote cloud storage 112, the migration/recallmanager 220 uses path statistics 700 to determine (at block 602) adegradation level on the mirror paths 110, as described with respect toFIGS. 7 and 8, which may be based on bandwidth utilization, number oflow link response time transfers, ratio of successful to unsuccessfultransfer, and whether the paths are in a high failure mode. If (at block604) the degradation level does not exceed a degradation criteria orthreshold, i.e., the mirror paths 110 are not sufficiently degraded,then the migration/recall manager 220 sends (at block 606) a command tothe primary storage server 200 _(P) to recall the extents from theremote cloud storage 112 to the primary storage 102 _(P). After recallby the primary server 200 _(P) only, the recalled extents are mirroredto the secondary storage 102 _(S) over mirror paths 110. If (at block604) the degradation level satisfies the degradation criteria, i.e., issufficiently degraded, then the migration/recall manager 220 sends (atblock 608) commands to both the primary 300 _(P) and secondary 300 _(S)servers to have each independently recall the extents from the remotestorage 112 to the primary 102 _(P) and secondary storages 102 _(S).

With the embodiment of FIG. 6, a determined degradation level on themirror paths 110 is used to determine whether to have each of theprimary 200 _(P) and secondary 200 _(S) servers recall the recallextents if the mirror paths 110 instead of recalling to the primaryserver 200 _(P) and then mirroring to the secondary server 200 _(S) onthe mirror paths 110. The described embodiments avoid using the mirrorpaths 110 if sufficiently degraded for recall. In this way, thedescribed embodiments allow for a faster recall if the second network114 is operating at higher speeds than the degraded mirror paths 110 andrecalling just from the second network 114 reduces bandwidth andutilization burdens on the mirror paths 110 while they are being used tomirror data from the primary server 200 _(P) to the secondary server 200_(S) in a degraded state.

FIG. 7 illustrates an embodiment of path statistics 700 maintained for amirror path in the mirror paths 110, and may include: a mirror pathidentifier (ID) 702, such as port ID, etc. for one of the mirror paths110; bandwidth utilization 704 indicating an amount or percentage ofbandwidth on the mirror path 702 being utilized; high response timetransfers 706 comprising a number of transfers of mirrored data on themirror path 702 from the primary server 200 _(P) to the secondary server200 _(S) that have a link response time within a range of high responsetimes, indicating poor path performance; low response time transfers 708comprising a number of transfers of mirrored data on the mirror path 702from the primary server 200 _(P) to the secondary server 200 _(S) havinglink response times within a range of low response times, indicatingacceptable/good path performance; successful transfers 710 comprising anumber of successful transfers of mirrored data on the mirror path 702;unsuccessful transfers 712 comprising a number of unsuccessful transfersof mirrored data on the mirror path 702; and indication of whether themirror path 702 is operating in a high failure rate mode 714, asseparately determined by a path manager, in which transfers on the path702 are throttled to recover the path 702 without suspending the path702 and during which performance is degraded on the path 702.

The path statistics 700 may be gathered periodically by a path manageror the operating system 208 during data mirroring operations on themirror paths 110, such as at predetermined intervals, after apredetermined number of transfers, etc.

FIG. 8 illustrates an embodiment of operations performed by themigration/recall manager 220 to determine whether the mirror paths 110are sufficiently degraded to have both the primary 200 _(P) andsecondary 200 _(S) storage servers separately recall extents. Uponinitiating (at block 800) an operation to determine whether mirror paths110 are degraded, the migration/recall manager 220 determines (at block802) whether overall mirror path bandwidth utilization 704 exceeds abandwidth threshold. If so, the mirror paths are categorized (at block804) as over utilized. If (at block 802) the paths are not overutilizedor after indicating (at block 804) paths as overutilized, for each linkresponse for each transfer made on a mirror path 110, a low responsetime transfers 708 is incremented (at block 806) in response to the linkresponse time for the transfer being within a low response time rangeand a high response time transfers 706 is incremented in response to thelink response time for the transfer being within a high response timerange. If (at block 808) the high response time transfers 706 exceeds athreshold number, then the mirror paths are categorized (at block 810)as having a high response time, which is undesirable.

From the no branch of block 808 or block 810, if (at block 812) a ratioof successful transfers 710 to unsuccessful transfers 712 exceeds athreshold ratio, then the mirror paths are categorized (at block 814) ashaving an unacceptable ratio of successful 710 to unsuccessful 712transfers. The migration/recall manager 220 determines (at block 816)whether mirror paths exceed a degradation criteria, indicatingsufficiently degraded, based on at least one of whether mirror paths 110are categorized as overutilized; having a high response time; having anunacceptable ratio of successful 710 to unsuccessful 712 transfers; andin high failure rate mode 714.

In making the determination of sufficiently degraded performance, themigration/recall manager 220 may require that all criteria indicatedegraded performance to determine that the mirror paths are degraded.Alternatively, the mirror paths may be determined to be degraded if lessthan all or just one of the criteria indicate degraded. In embodimentswhere there are multiple mirror paths, a determination that the mirrorpaths as a whole are degraded may require that the degradation criteriaindicating degradation, such as bandwidth utilization, high responsetime, unacceptable ratio of successful transfers, and high failure ratemode, is determined for all the mirror paths, some predetermined numberor percentage of paths or for just one path.

With the embodiment of FIG. 8, multiple criteria concerning mirror pathfunctioning, such as bandwidth utilization, high response time,unacceptable ratio of successful transfers, and high failure rate mode,determined for the different mirror paths may be used to determinewhether the mirror paths, comprising one or more paths, are degraded toan extent that recall extents should be recalled from the remote storage112 to both the primary 100 _(P) and secondary 100 _(S) storage systemsto avoid using the degraded mirror paths 110.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The computational components of FIG. 1, including the servers 200 _(P),200 _(S), 200 _(H), and remote cloud storage 112 may be implemented inone or more computer systems, such as the computer system 902 shown inFIG. 9. Computer system/server 902 may be described in the generalcontext of computer system executable instructions, such as programmodules, being executed by a computer system. Generally, program modulesmay include routines, programs, objects, components, logic, datastructures, and so on that perform particular tasks or implementparticular abstract data types. Computer system/server 902 may bepracticed in distributed cloud computing environments where tasks areperformed by remote processing devices that are linked through acommunications network. In a distributed cloud computing environment,program modules may be located in both local and remote computer systemstorage media including memory storage devices.

As shown in FIG. 9, the computer system/server 902 is shown in the formof a general-purpose computing device. The components of computersystem/server 902 may include, but are not limited to, one or moreprocessors or processing units 904, a system memory 906, and a bus 908that couples various system components including system memory 906 toprocessor 904. Bus 908 represents one or more of any of several types ofbus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, andnot limitation, such architectures include Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnects (PCI) bus.

Computer system/server 902 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 902, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 906 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 910 and/or cachememory 912. Computer system/server 902 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 913 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 908 by one or more datamedia interfaces. As will be further depicted and described below,memory 906 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 914, having a set (at least one) of program modules 916,may be stored in memory 906 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. The components of the computer 902 may be implemented asprogram modules 916 which generally carry out the functions and/ormethodologies of embodiments of the invention as described herein. Thesystems of FIG. 1 may be implemented in one or more computer systems902, where if they are implemented in multiple computer systems 902,then the computer systems may communicate over a network.

Computer system/server 902 may also communicate with one or moreexternal devices 918 such as a keyboard, a pointing device, a display920, etc.; one or more devices that enable a user to interact withcomputer system/server 902; and/or any devices (e.g., network card,modem, etc.) that enable computer system/server 902 to communicate withone or more other computing devices. Such communication can occur viaInput/Output (I/O) interfaces 922. Still yet, computer system/server 902can communicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 924. As depicted, network adapter 924communicates with the other components of computer system/server 902 viabus 908. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 902. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s)” unless expressly specifiedotherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the present inventionneed not include the device itself.

The foregoing description of various embodiments of the invention hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed. Many modifications and variations are possible in lightof the above teaching. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto. The above specification, examples and data provide acomplete description of the manufacture and use of the composition ofthe invention. Since many embodiments of the invention can be madewithout departing from the spirit and scope of the invention, theinvention resides in the claims herein after appended.

What is claimed is:
 1. A computer program product for migrating mirroredextents at a primary storage and a secondary storage to a third storage,wherein the computer program product comprises a computer readablestorage medium having program instructions executable by a processor tocause operations, the operations comprising: determining accessstatistics with respect to mirrored extents of data at the primarystorage mirrored to the secondary storage to migrate to the thirdstorage; migrating a first set of the mirrored extents associated withaccess statistics indicating a highest level of access of the mirroredextents from the secondary storage to the third storage; and migrating asecond set of the mirrored extents associated with access statisticsindicating a lower level of access than the mirrored extents in thefirst set from the primary storage to the third storage.
 2. The computerprogram product of claim 1, wherein more Input/Output (I/O) requestsfrom hosts are directed to the primary storage than to the secondarystorage, and wherein the third storage comprises cloud based storageaccessible to a primary server managing access to the primary storageand a secondary server managing access to the secondary storage.
 3. Thecomputer program product of claim 1, wherein the operations furthercomprise: determining whether the mirrored extents to migrate to thethird storage exceed an amount threshold; and migrating the mirroredextents from the primary storage to the third storage in response todetermining that the mirrored extents to migrate do not exceed theamount threshold, wherein the determining the access statistics,migrating the first set of the mirrored extents, and migrating thesecond set of the mirrored extents are performed in response todetermining that the mirrored extents to migrate exceed the amountthreshold.
 4. The computer program product of claim 1, wherein an accessstatistic for each mirrored extent of the mirrored extents comprises adelta of a primary access statistic, for the mirrored extent at theprimary storage, and a secondary access statistic, for the mirroredextent at the secondary storage, wherein the first set of the mirroredextents have greater deltas than the second set of the mirrored extents.5. The computer program product of claim 1, wherein the mirrored extentsare configured in ranks in the primary storage and the secondarystorage, wherein an access statistic for each mirrored extent of themirrored extents comprises a delta of a primary access statistic, for aprimary rank at the primary storage, including the mirrored extent, anda secondary access statistic, for a secondary rank at the secondarystorage, including the mirrored extent wherein the first set of themirrored extents have greater deltas than the second set of the mirroredextents.
 6. The computer program product of claim 1, wherein thedetermining the access statistics comprises: for each mirrored extent ofthe mirrored extents, determining a delta between a primary readlatency, for the mirrored extent on the primary storage, and a secondaryread latency, for the mirrored extent on the secondary storage, whereinthe first set of the mirrored extents has highest deltas of the deltasdetermined for the mirrored extents and wherein the second set of themirrored extents has lower deltas than the deltas in the first set ofthe mirrored extents.
 7. The computer program product of claim 1,wherein the determining the access statistics comprises: for eachmirrored extent of the mirrored extents, determining a delta between aprimary read activity, for the mirrored extent on the primary storage,and a secondary read activity, for the mirrored extent on the secondarystorage, wherein the first set of the mirrored extents has highestdeltas of the deltas determined for the mirrored extents and wherein thesecond set of the mirrored extents has lower deltas than the deltas inthe first set of the mirrored extents.
 8. The computer program productof claim 1, wherein the determining the access statistics comprises: foreach mirrored extent of the mirrored extents, determining a deltabetween a primary Input/Output (I/O) access activity at a primary rankincluding the mirrored extent on the primary storage and a secondary I/Oaccess activity at a secondary rank including the mirrored extent on thesecondary storage, wherein the first set of the mirrored extents hashighest deltas of the deltas determined for the mirrored extents andwherein the second set of the mirrored extents has lower deltas than thedeltas in the first set of the mirrored extents.
 9. The computer programproduct of claim 1, wherein the determining the access statisticscomprises: for each mirrored extent of the mirrored extents, performing:determining at least two of: a read latency difference for the mirroredextent on the primary and the secondary storages; a read activitydifference for the mirrored extent on the primary and the secondarystorages; and a rank I/O activity difference comprising a difference ofactivity at a primary rank including the mirrored extent on the primarystorage and a secondary rank including the mirrored extent on thesecondary storage; and determining an aggregate activity score based ona weighted aggregation of at least two of the read latency difference,the read activity difference, and the rank I/O activity difference,wherein the first set of the mirrored extents has highest total scoresfor the mirrored extents and wherein the second set of the mirroredextents has lower total scores than the total scores in the first set ofthe mirrored extents.
 10. A system for migrating mirrored extents at aprimary storage and a secondary storage to a third storage, comprising:a processor; and a computer readable storage medium having programinstructions that when executed by the processor to cause operations,the operations comprising: determining access statistics with respect tomirrored extents of data at the primary storage mirrored to thesecondary storage to migrate to the third storage; migrating a first setof the mirrored extents associated with access statistics indicating ahighest level of access of the mirrored extents from the secondarystorage to the third storage; and migrating a second set of the mirroredextents associated with access statistics indicating a lower level ofaccess than the mirrored extents in the first set from the primarystorage to the third storage.
 11. The system of claim 10, wherein anaccess statistic for each mirrored extent of the mirrored extentscomprises a delta of a primary access statistic, for the mirrored extentat the primary storage, and a secondary access statistic, for themirrored extent at the secondary storage, wherein the first set of themirrored extents have greater deltas than the second set of the mirroredextents.
 12. The system of claim 10, wherein the determining the accessstatistics comprises: for each mirrored extent of the mirrored extents,determining a delta between a primary read latency, for the mirroredextent on the primary storage, and a secondary read latency, for themirrored extent on the secondary storage, wherein the first set of themirrored extents has highest deltas of the deltas determined for themirrored extents and wherein the second set of the mirrored extents haslower deltas than the deltas in the first set of the mirrored extents.13. The system of claim 10, wherein the determining the accessstatistics comprises: for each mirrored extent of the mirrored extents,determining a delta between a primary read activity, for the mirroredextent on the primary storage, and a secondary read activity, for themirrored extent on the secondary storage, wherein the first set of themirrored extents has highest deltas of the deltas determined for themirrored extents and wherein the second set of the mirrored extents haslower deltas than the deltas in the first set of the mirrored extents.14. The system of claim 10, wherein the determining the accessstatistics comprises: for each mirrored extent of the mirrored extents,determining a delta between a primary Input/Output (I/O) access activityat a primary rank including the mirrored extent on the primary storageand a secondary I/O access activity at a secondary rank including themirrored extent on the secondary storage, wherein the first set of themirrored extents has highest deltas of the deltas determined for themirrored extents and wherein the second set of the mirrored extents haslower deltas than the deltas in the first set of the mirrored extents.15. The system of claim 10, wherein the determining the accessstatistics comprises: for each mirrored extent of the mirrored extents,performing: determining at least two of: a read latency difference forthe mirrored extent on the primary and the secondary storages; a readactivity difference for the mirrored extent on the primary and thesecondary storages; and a rank I/O activity difference comprising adifference of activity at a primary rank including the mirrored extenton the primary storage and a secondary rank including the mirroredextent on the secondary storage; and determining an aggregate activityscore based on a weighted aggregation of at least two of the readlatency difference, the read activity difference, and the rank I/Oactivity difference, wherein the first set of the mirrored extents hashighest total scores for the mirrored extents and wherein the second setof the mirrored extents has lower total scores than the total scores inthe first set of the mirrored extents.
 16. A method for migratingmirrored extents at a primary storage and a secondary storage to a thirdstorage, comprising: determining access statistics with respect tomirrored extents of data at the primary storage mirrored to thesecondary storage to migrate to the third storage; migrating a first setof the mirrored extents associated with access statistics indicating ahighest level of access of the mirrored extents from the secondarystorage to the third storage; and migrating a second set of the mirroredextents associated with access statistics indicating a lower level ofaccess than the mirrored extents in the first set from the primarystorage to the third storage.
 17. The method of claim 16, wherein anaccess statistic for each mirrored extent of the mirrored extentscomprises a delta of a primary access statistic, for the mirrored extentat the primary storage, and a secondary access statistic, for themirrored extent at the secondary storage, wherein the first set of themirrored extents have greater deltas than the second set of the mirroredextents.
 18. The method of claim 16, wherein the determining the accessstatistics comprises: for each mirrored extent of the mirrored extents,determining a delta between a primary read latency, for the mirroredextent on the primary storage, and a secondary read latency, for themirrored extent on the secondary storage, wherein the first set of themirrored extents has highest deltas of the deltas determined for themirrored extents and wherein the second set of the mirrored extents haslower deltas than the deltas in the first set of the mirrored extents.19. The method of claim 16, wherein the determining the accessstatistics comprises: for each mirrored extent of the mirrored extents,determining a delta between a primary read activity, for the mirroredextent on the primary storage, and a secondary read activity, for themirrored extent on the secondary storage, wherein the first set of themirrored extents has highest deltas of the deltas determined for themirrored extents and wherein the second set of the mirrored extents haslower deltas than the deltas in the first set of the mirrored extents.20. The method of claim 16, wherein the determining the accessstatistics comprises: for each mirrored extent of the mirrored extents,performing: determining at least two of: a read latency difference forthe mirrored extent on the primary and the secondary storages; a readactivity difference for the mirrored extent on the primary and thesecondary storages; and a rank I/O activity difference comprising adifference of activity at a primary rank including the mirrored extenton the primary storage and a secondary rank including the mirroredextent on the secondary storage; and determining an aggregate activityscore based on a weighted aggregation of at least two of the readlatency difference, the read activity difference, and the rank I/Oactivity difference, wherein the first set of the mirrored extents hashighest total scores for the mirrored extents and wherein the second setof the mirrored extents has lower total scores than the total scores inthe first set of the mirrored extents.